Fix keep_max_files re-download churn by suppressing pruned entries - #1487
Fix keep_max_files re-download churn by suppressing pruned entries#1487michaeldyrynda wants to merge 2 commits into
Conversation
…ad archive Entries pruned by keep_max_files were fully removed from the download archive, causing yt-dlp to re-download them on every run. Now pruned entries are marked as suppressed — files are deleted but the entry ID stays in the archive to prevent re-download churn. - Add suppressed flag to DownloadMapping with backward-compatible serialization - Add suppress_entry/unsuppress paths in DownloadMappings - Use _suppress_entry for keep_max_files pruning, _remove_entry for date_range - Filter suppressed entries from keep_max_files counting loop
|
Another solution to consider is, if a playlist-based sort is used, we could add a playlist_index parameter under the hood. I.e., keep_max is 20, using a playlist idx ascending, we could pass 0:20 to only look at that subset of videos. I think that could be more clean than wrangling the archive more. Its also midnight and Im in bed lol, will need to think about this a bit more. I do appreciate the quick follow-up! |
|
That does make sense. I'll let you think a bit more on how you want it implemented and I can take another crack. All good on the quick follow, i've been downloading the same 50 videos for a few days since the update so it's driving me nuts in the meantime ha |
|
In your setup, you should try passing: or assuming max to keep is 20, and see if that solves things |
|
Alright, i think that worked. Added a new preset to # config.yaml
presets:
"Last N Playlist Items":
ytdl_options:
playlist_items: "-{only_recent_max_files}:"
# subscriptions.yaml
'Plex TV Show by Date | Only Recent | Last N Playlist Items':
'= Tech':
'~AI that works':
url: 'https://www.youtube.com/playlist?list=PLi60mUelRAbFqfgymVfZttlkIyt0XHZjt'
only_recent_max_files: 50Only caveat this way is needing to create two presets if you wanted to grab the |
|
This line is where we can inject ytdl_option args based on output options: https://github.com/jmbannon/ytdl-sub/blob/master/src/ytdl_sub/subscriptions/subscription_ytdl_options.py#L97 Based on the sort, we can pass the appropriate value. |
2 participants
Summary
Follow-up to #1484 / #1461.
Entries pruned by
keep_max_fileswere fully removed from the download archive, causing yt-dlp to re-download them on every run - only for them to be immediately pruned again in an endless cycle.This fix introduces a
suppressedflag on pruned entries: files are deleted from disk, but the entry ID stays in the download archive so yt-dlp knows not to re-download it.Changes
suppressedflag onDownloadMapping- backward-compatible serialisation (old archives without the field default toFalse)suppress_entry()/_suppress_entry()- clears file names and marks the entry as suppressed, keeping it in the archiveremove_stale_files()- uses_suppress_entryforkeep_max_filespruning (date-range pruning still fully removes entries, since yt-dlp's own date filter prevents re-download)add_entry()- unsuppresses if a previously-suppressed entry gets re-downloadedkeep_max_filescounting so they don't consume slotsTests
keep_max_filessort tests to verify both active and suppressed entry setsTestDownloadMappingSuppressedfor serialisation roundtripTestSuppressedEntriesPreventRedownloadcovering: archive inclusion, slot counting, file deletion, and JSON roundtrip